Bayesian Statistics

The missing part of the puzzle

"Floating jigsaw pieces, Säveån" by Eva the Weaver is licensed under CC BY-NC-SA 2.0

We have all had the experience of searching for something online; and then logging onto our social media accounts and seeing strangely personalized adverts. What few people realize is that the algorithms that generate these adverts were developed using mathematical principles first discovered in the 18th Century. The approach is called Bayesian statistics and it has slowly been taking over the world since the 1940s. Bayesians won the Second World War for the Allies; established the link between smoking and cancer; and develop personalized content on the internet.

If you’ve never heard of Bayesian statistics or you know very little about the field, don’t worry. The field deserves to be much better known by the general public. The details might get a little technical, but the principles are very simple.

Bayesian statistics take their name from Thomas Bayes, who studied logic and theology at the University of Edinburgh. He developed a revolutionary approach to statistics. It starts with a guess and then adjusts the result as more evidence comes in. Critics hated it, so he didn’t publish it in his lifetime. His friend Richard Price discovered it in his papers after his death in 1761 and published the first version of the formula. Meanwhile, a young French mathematician called Pierre-Simon Laplace independently discovered elements of the approach. Price told him about the role of guesses and Laplace then published an elegant version of the formula at the beginning of the 19th century.

Academic mathematicians hated the Bayesian approach for many years, but it was adopted by artillery officers, actuaries and biologists and a few other practical people. Alan Turing’s team used Bayesian techniques to crack the Nazis’ Enigma code in the Second World War; and the approach has been adopted enthusiastically by software engineers since the 1970s since Turing’s wartime work was declassified.

So, how do Bayesian statistics work? Think about sailors lost at sea. The navigator might make a random guess about the ship’s position and then compare the results of the guess with the stars above. The difference between the guess and the real result allows the navigator to make a second guess about the ship’s position. Another comparison with the real position can help refine the result. The navigator can make better and better guesses until finally the result of the guess coincides with the ship’s actual position. The process can be repeated indefinitely.

The key to a deeper understanding of Bayesian statistics is that it implies that all our knowledge is provisional. We might think we know something, but what if we are wrong? Bayesians will seek to adjust their beliefs as more evidence is unveiled. Bayesians often think of a sliding scale from 0% to 100% and move their beliefs up or down the scale depending on the available evidence.

The beauty of the Bayesian approach is that it acts as the opposite of cognitive dissonance - the natural tendency we all have to double down on our starting position when confronted by contradictory evidence. It is for this reason that I consider Bayesian statistics to be the missing piece in the puzzle for people who want to do their own research about the world.

If we compare our brains to a smartphone, Bayesian statistics can act as a mental app to help us hold our opinions lightly as we mull the evidence. I discuss this in much greater depth in Chapter Three of Sharpen Your Axe, my free book on critical-thinking tools and techniques. I also mention two other tools - the prisoner’s dilemma and Ockham’s razor - which we could see as extensions to make the app work better. If you missed the beginning of the book, you can find the beta versions of Chapter One here and Chapter Two here.

I hope you have enjoyed this post. If you haven’t subscribed yet, please do so so you don’t miss any updates. I have decided to make the book free in order to spread it as widely as possible. It would be greatly appreciated if you could share it on social media or with any friends who might appreciate the content. The next installment will come on Tuesday 5th January and will discuss how our gut judgements can lead us astray. See you then!

Update (25 April 2021)

The full beta version is available here

[Updated on 10 March 2022] Opinions expressed on Substack and Twitter are those of Rupert Cocke as an individual and do not reflect the opinions or views of the organization where he works or its subsidiaries.